Church Related Information Technology Blog

This blog is to help make logic of backups for evaluation purposes. It will be a place to direct management and staff that have questions about our procedure.

Hardware

CCN uses a Quantum DLT-V4 dual tape drive rack mount for tape backups. We use Quantum DLTtape VS1 cartridges which will backup 160/320 GB with our DLT-V4 tape drives.

The drives are connected to two of our three HP ProLiant DL380 G5 servers. cc-nash-dc-02 which is our primary domain controller, file server and print server. cc-nash-exch-02 which is our Exchange 2007 mail server.

Backup Software

We are currently using Symantec BackupExec version 12.5.  The plan is to install the 2010 version soon.
Update: Installed BE2010 on May 28th, 2010 on all servers.

We purchased two copies of the version for Windows Servers, a remote server backup agent and an Exchange agent. We also renew our maintenance subscription yearly on these products.

The third server, cc-nash-sql-02, is backed up by BackupExec installed on cc-nash-exch-02 using BackupExec’s Remote Server agent. We also have one virtual server cc-nash-vs2-02 whose existing folder gets backed up at this time. Its functions are Blackberry Enterprise Server and Spiceworks Server. I am not sure currently if a restore is possible with the files that get backed up for the virtual server, however no real pertinant data is contained in it.

Brick level backups are performed on the Exchange Mail Server, meaning that we can restore down to a single item (as in a single email or contact).

Capacity

cc-nash-dc-02

Six 72GB 15k rpm SAS drives / RAID 5 / 1 hot spare (5 useable drives)

Total Capacity = 273.34 GB

c drive (OS)     – total capacity   39 GB – current used    23 GB

e drive (Data) – total capacity 234 GB – current used 186 GB

Total being backed up nightly = 209 GB

cc-nash-exch-02

Six 72GB 15k rpm SAS drives / RAID 5 / 1 hot spare (5 useable drives)

Total Capacity = 273.34 GB

c drive (OS)     – total capacity   39 GB – current used    33.3 GB

e drive (Data) – total capacity 234 GB – current used 46.6 GB

Total being backed up nightly = 79.9 GB

cc-nash-sql-02

Six 72GB 15k rpm SAS drives / RAID 5 / 1 hot spare (5 useable drives)

Total Capacity = 273.34 GB

c drive (OS)     – total capacity   39 GB – current used    21.6 GB

e drive (Data) – total capacity 234 GB – current used 154 GB (this includes the cc-nash-vs2-02 virtual server and large file storage for media department)

Total being backed up nightly = 171.6 GB

CCN Backup Cycle

CCN’s backup cycle uses 10 tapes for a 2 week rotation period of full nightly backups. There are two sets of tapes consisting of 5 tapes each for each day of the week Monday thru Friday. Set one is labeled A and set two is labeled B. Monday A, Tuesday A, Wednesday A, etc.

At the end of the month, whatever tapes are nearest to end of the month are removed, labeled by month end/year and that tape is replaced by a new tape. The most current months backup tape is placed in a fire proof safe in my office along with the previous years matching month. All other backup tapes are kept in a cabinet in the server room.

With the above procedure, we are able to restore any given file for any day within a two week period. We can also restore any given file for any month for any year. The cost of the procedure is two tapes per month currently. While we could start to put the tapes back into the cycle after a given amount of time, the thought is that the tapes are getting replaced prior to wearing out.  *note to make a plan to identify tapes date put into use in order to make sure they are rotating out effectively – a label would work

Currently backup tapes are rarely being taken off site due to questionable yet potential security risks. We MUST come up with an agreeable plan for this ASAP! Disaster recovery would be impossible with out off site backups.

Problems recognized with plan – if a user creates a file and deletes it prior to the end of the day, then no backup will be performed. The same is true of Exchange mailbox items. Need to come up with a Continuous Backup plan.

Disk to Disk backup

Was backing up incrementals to an external usb drive. The drive had mechanical problems and has been removed. Hope to replace it.
Update: Replaced faulty external drive with a Toshiba 1tb external usb drive that is also capable of eSATA.

Disk to Disk to Tape backup

This may be something to look at because theoretically backing up disk to disk should go fast, then backing up to tape wouldn’t affect the servers processing power and could back up during the day or when ever.
Update: tried linking a job to duplicate the backup to the external drive to the disk when it finishes but the job fills the tape. Suspect due to no compression on the duplicate job.

Online Backup

Created an account with Jungle Disk and testing now

What is backed up?

Workstations are set to redirect “My Documents” folders to the server. This has been a life saver in many cases but can be problematic when users put pictures, movies, etc in the folders. Also, by default iTunes puts all data in the My Music folder and this has to be manually changed. I have set the backup to exclude files with the .mp3 extensions to keep the backup from requiring more than one tape.

The file server cc-nash-dc-02 is backed up to tape every weeknight starting at 11:59 PM

Our bookstore server’s booklog database is backed up nightly to the cc-nash-dc-02 server at E:\Backups\booklog using Windows Scheduler on the Booklog server. This is set to occur prior to the nightly tape backup on cc-nash-dc-02.

The mail server cc-nash-exch-02 is backed up nightly at 11:45PM

The SQL server cc-nash-sql-02 is backed up nightly thru the wire during the cc-nash-exch-02 backup job.

Our Shelby Systems database is backed up by using 7-zip instead of the Shelby Database Backup. This was changed due to the fact that many times the Shelby Database Backup simply quits working and will not work at all once the database has reached a certain size. Here is the 7-zip bat file script: (make sure you understand what this does and test it before you put it to use … it is set to delete files so they dont pile up)

FOR /F “tokens=*” %%A in (‘dir /b *.bak’) DO CALL :BackupRoutine %%A
GOTO:EOF

:BackupRoutine
            echo %*
            set file=%*
            set noext=%file:~0,-4%
            E:\Shelby~2\7z.exe a %noext%.zip “E:\Shelby~2\%noext%.bak” -pyourpasswordgoeshere
     DEL E:\Shelby~2\%noext%.bak \y
GOTO:EOF

The Parent Pager database is backed up nightly using the SQL Server 2005 Management Studio using management–> maintenance plans. Occurs every week on Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday at 11:59:00 PM. Schedule will be used starting on 2/20/2009.

Future Plans

Disaster Recovery – Future plans should include remote failover servers. Perhaps a clustering server at the white house on the lot or maybe something more distant for disaster recovery purposes should be evaluated. At the very least, we should have an extra server that could be ready to go at any time should we experience a disaster. Waiting for hardware would waste a lot of valuable time and many times it is impossible to get identical hardware, especially if your hardware is no longer existant! It has been my experience that servers of the same brand and model do not always contain exact hardware also (chipsets, etc.)

Virutal servers should be part of the future plans. I particularly like VMWare ESXi because the host is not OS dependant and the bare metal restore ability.  Also, for a single host with unlimited virtual servers for free, its hard to beat. VMWares VShere 4 sounds like its the way to go once you get your virtualization feet wet. I haven’t ventured to Microsoft’s HyperV solution yet, primarily because I really like not relying on a base install OS.